Depth sensing with depth-adaptive illumination

ABSTRACT

An adaptive depth sensing system (ADSS) illuminates a scene with a pattern that is constructed based on an analysis of at least one prior-generated depth map. In one implementation, the pattern is a composite pattern that includes two or more component patterns associated with different depth regions in the depth map. The composite pattern may also include different illumination intensities associated with the different depth regions. By using this composite pattern, the ADSS can illuminate different objects in a scene with different component patterns and different illumination intensities, where those objects are located at different depths in the scene. This process, in turn, can reduce the occurrence of defocus blur, underexposure, and overexposure in the image information.

BACKGROUND

A conventional structured light depth sensing system operates byprojecting a fixed 2D pattern onto a scene. The depth sensing systemthen captures image information which represents the scene, asilluminated by the pattern. The depth sensing system then measures theshift that occurs between the original pattern that is projected ontothe scene and pattern content that appears in the captured imageinformation. The depth sensing system can then use this shift, togetherwith the triangulation principle, to determine the depth of surfaces inthe scene.

A depth sensing system may produce image information having poor qualityin certain circumstances. However, known depth sensing systems do notaddress these quality-related issues in a satisfactory manner.

SUMMARY

An adaptive depth sensing system (ADSS) is described herein whichproduces image information with improved quality (with respect tonon-adaptive depth sensing systems). For example, the ADSS can reducethe occurrence of defocus blur, overexposure, and underexposure in theimage information captured by the ADSS. The ADSS achieves this result byilluminating a scene with a pattern that is constructed based on ananalysis of at least the last-generated depth map.

In one implementation, the ADSS operates by identifying different depthregions in the depth map(s). The ADSS then generates a composite patternhaving different component patterns and illumination intensitiesassigned to the respective depth regions. Each component patternincludes features that have a particular property, and differentcomponent patterns include features having different respectiveproperties. For example, in one non-limiting case, each componentpattern includes features having a particular size and/or illuminationintensity, and different component patterns include features havingdifferent respective sizes and/or illumination intensities.

The above approach can be manifested in various types of systems,components, methods, computer readable storage media, data structures,articles of manufacture, and so on.

This Summary is provided to introduce a selection of concepts in asimplified form; these concepts are further described below in theDetailed Description. This Summary is not intended to identify keyfeatures or essential features of the claimed subject matter, nor is itintended to be used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an illustrative environment that includes an adaptive depthsensing system (ADSS).

FIG. 2 graphically illustrates one manner by which the ADSS of FIG. 1can generate a composite pattern.

FIG. 3 is a flowchart that represents an overview of one manner ofoperation of the ADSS of FIG. 1.

FIG. 4 is a flowchart that represents a more detailed explanation of onemanner of operation of the ADSS of FIG. 1.

FIG. 5 shows functionality for generating a set of component patterns.

FIG. 6 is a flowchart that represents one way of generating the set ofcomponent patterns.

FIG. 7 is a flowchart that represents one approach for deriving amapping table. The mapping table maps different depth ranges todifferent respective component patterns.

FIG. 8 shows a lens and ray diagram associated with a projector deviceand a camera device which interact with the ADSS. This diagram is usedto derive a blur model.

FIG. 9 shows a relationship between depth and diameter of blur. Thisrelationship is used to determine parameters of the blur model.

FIG. 10 shows relationships between depth and the matching accuracy,with respect to different component patterns. These relationships areproduced by inducing blur in component patterns using the blur model,and comparing the un-blurred component patterns with the blurredcomponent patterns.

FIG. 11 shows one representative mapping table which maps differentdepth ranges to different respective component patterns. This mappingtable is produced, in part, based on analysis of the relationships ofFIG. 10.

FIG. 12 shows illustrative computing functionality that can be used toimplement any aspect of the features shown in the foregoing drawings.

The same numbers are used throughout the disclosure and figures toreference like components and features. Series 100 numbers refer tofeatures originally found in FIG. 1, series 200 numbers refer tofeatures originally found in FIG. 2, series 300 numbers refer tofeatures originally found in FIG. 3, and so on.

DETAILED DESCRIPTION

This disclosure is organized as follows. Section A describes an overviewof an illustrative adaptive depth sensing system (ADSS). Section Bdescribes illustrative functionality for deriving a set of componentpatterns for use by the ADSS. Section C describes an illustrativeapproach for generating a mapping table for use by the ADSS. Section Ddescribes an illustrative approach for setting up and initializing theADSS. Section E describes illustrative computing functionality that canbe used to implement any aspect of the features described in precedingsections.

As a preliminary matter, some of the figures describe concepts in thecontext of one or more structural components, variously referred to asfunctionality, modules, features, elements, etc. The various componentsshown in the figures can be implemented in any manner by any physicaland tangible mechanisms, for instance, by software, hardware (e.g.,chip-implemented logic functionality), firmware, etc., and/or anycombination thereof. In one case, the illustrated separation of variouscomponents in the figures into distinct units may reflect the use ofcorresponding distinct physical and tangible components in an actualimplementation. Alternatively, or in addition, any single componentillustrated in the figures may be implemented by plural actual physicalcomponents. Alternatively, or in addition, the depiction of any two ormore separate components in the figures may reflect different functionsperformed by a single actual physical component. FIG. 12, to bediscussed in turn, provides additional details regarding oneillustrative physical implementation of the functions shown in thefigures.

Other figures describe the concepts in flowchart form. In this form,certain operations are described as constituting distinct blocksperformed in a certain order. Such implementations are illustrative andnon-limiting. Certain blocks described herein can be grouped togetherand performed in a single operation, certain blocks can be broken apartinto plural component blocks, and certain blocks can be performed in anorder that differs from that which is illustrated herein (including aparallel manner of performing the blocks). The blocks shown in theflowcharts can be implemented in any manner by any physical and tangiblemechanisms, for instance, by software, hardware (e.g., chip-implementedlogic functionality), firmware, etc., and/or any combination thereof.

As to terminology, the phrase “configured to” encompasses any way thatany kind of physical and tangible functionality can be constructed toperform an identified operation. The functionality can be configured toperform an operation using, for instance, software, hardware (e.g.,chip-implemented logic functionality), firmware, etc., and/or anycombination thereof.

The term “logic” encompasses any physical and tangible functionality forperforming a task. For instance, each operation illustrated in theflowcharts corresponds to a logic component for performing thatoperation. An operation can be performed using, for instance, software,hardware (e.g., chip-implemented logic functionality), firmware, etc.,and/or any combination thereof. When implemented by a computing system,a logic component represents an electrical component that is a physicalpart of the computing system, however implemented.

The phrase “means for” in the claims, if used, is intended to invoke theprovisions of 35 U.S.C. §112, sixth paragraph. No other language, otherthan this specific phrase, is intended to invoke the provisions of thatportion of the statute.

The following explanation may identify one or more features as“optional.” This type of statement is not to be interpreted as anexhaustive indication of features that may be considered optional; thatis, other features can be considered as optional, although not expresslyidentified in the text. Finally, the terms “exemplary” or “illustrative”refer to one implementation among potentially many implementations.

A. Overview

FIG. 1 shows an illustrative environment that includes an adaptive depthsensing system (ADSS) 102. The ADSS 102 uses a projector device 104 toproject a pattern onto a scene 106. The ADSS 102 uses a camera device108 to capture image information from the scene 106. That is, the imageinformation represents the scene 106 as illuminated by the pattern. TheADSS 102 then analyzes the image information, in conjunction with theoriginal pattern, to construct a depth map. The depth map represents thedepths of surfaces within the scene 106.

For example, in the representative case of FIG. 1, the depth maprepresents the depths of the surfaces of a sphere 110 and pyramid 112within the scene 106. The depths are measured with respect to areference point, such as the source of the projected pattern (e.g., theprojector device 104). In this representative example, the sphere 110 islocated at an average distance d₁ with respect to the reference point,while the pyramid 112 is located at an average distance d₂ with respectto the reference point, where d₁<d₂.

The ADSS 102 is adaptive in the sense that the manner in which itilluminates the scene 106 is dependent on its analysis of at least thelast-computed depth map. More formally stated, the ADSS 102 performsanalysis on at least depth map D_(t−1) that has been produced at timet−1, to derive an analysis result. The ADSS 102 then generates a patternP_(t) based on this analysis result. In doing so, the ADSS 102 canprovide appropriate illumination to different surfaces in the scene 106which occur at different respective depths. For example, the ADSS 102can illuminate the sphere 110 with one part of the pattern P_(t) andilluminate the pyramid 112 with another part of the pattern P_(t). Thismanner of operation, in turn, can improve the quality of the imageinformation captured by the camera device 108. For example, the ADSS 102can reduce the occurrence of defocus blur, underexposure, and/oroverexposure in the image information.

With the above introduction, the components of FIG. 1 will be nowdescribed in detail. The ADSS 102 as a whole can be implemented by oneor more computing devices of any type, such as a personal computer, atablet computing device, a mobile telephone device, a personal digitalassistant (PDA) device, a set-top box, a game console device, and so on.More generally, the functionality of the ADSS 102 can be contained at asingle site or distributed over plural sites. For example, although notshown, one or more components of the ADSS 102 can rely on processingfunctionality that is provided at a remote location with respect to theprojector device 104 and the camera device 108. The ADSS 102 can alsoinclude one or more graphical processing units that perform parallelprocessing of image information to speed up the operation of the ADSS102. Section E provides further details regarding representativeimplementations of the ADSS 102.

The ADSS 102 is coupled to the projector device 104 and the cameradevice 108 through any type of communication conduits (e.g., hardwiredlinks, wireless links, etc.). Alternatively, the projector device 104and/or the camera device 108 may correspond to components in the samehousing as the ADSS 102. The projector device 104 can be implemented asa programmable projector that projects a pattern onto a scene. In onemerely representative case, for example, the projector device 104 can beimplemented by the DLP® LightCommander™ produced by Texas Instruments,Incorporated, of Dallas, Tex., or any like technology. The camera device108 can be implemented by any type of image capture technology, such asa CCD device. The projector device 104 and camera device 108 can emitand receive, respectively, any type of electromagnetic radiation, suchas visible light, infrared radiation, etc.

While not illustrated in FIG. 1, in one representative setup, a lens ofthe projector device 104 can be vertically aligned with a lens of thecamera device 108. (This is illustrated, for example, in FIG. 8, to bedescribed in turn.). The vertical arrangement of the projector device104 with respect to the camera device 108 ensures that pattern shiftonly occurs vertically. However, other implementations can adopt otherorientations of the projector device 104 with respect to the cameradevice 108. Moreover, other implementations can include two or moreprojector devices (not shown) and/or two or more camera devices (notshown). For example, the use of two or more camera devices allows theADSS 102 to provide a more comprehensive representation of objectswithin the scene 106.

The projector device 104 may also include a synchronization module 114.The synchronization module 114 synchronizes the operation of theprojector device 104 with the camera device 108. For example, thesynchronization module 114 can prompt the camera device 108 to captureimage information in response to the projection of a pattern by theprojector device 104, e.g., by sending a triggering signal from theprojector device 104 to the camera device 108. In other implementations,the synchronization module 114 may represent standalone functionality(that is, functionality that is separate from the projector device 104),or functionality that is incorporated into some other module shown inFIG. 1.

The ADSS 102 itself can include (or can be conceptualized as including)multiple components that perform different tasks. A depth determinationmodule 116 utilizes the principle of triangulation to generate a depthmap of the scene 106, based on the original pattern P_(t) that isprojected onto the scene 106, and the pattern that appears in the imageinformation captured by the camera device 108. More formally stated, thedepth determination module 116 first determines a pattern shift Δ_(t)that occurs between image information I_(t) captured by the cameradevice 108 at time t, and a reference image I_(t)′. (Additionalexplanation will be provided below regarding the manner in which thereference image I_(t)′ is computed.) For example, the depthdetermination module 116 can produce the pattern shift Δ_(t) by matchingeach pixel in an extended block in the captured image information I_(t)with its most similar counterpart in the reference image I_(t)′. Thedepth determination module 116 can execute this matching in any manner,such as by using a normalized cross-correlation technique.

The depth determination module 116 can then determine the depth map,D_(t), based on the equation:

${D_{t} = \frac{{FHD}_{0}}{{FH} + {D_{0}\Delta_{t}}}},{t = 1},2,{\ldots \mspace{14mu}.}$

In this expression, F is the focal length of the camera device 108, H isthe distance between projector device's lens and the camera device'slens, D₀ refers to a depth that is used in connection with generatingthe reference image I_(t)′ (to be described below), and Δ_(t) is thepattern shift. In one representative implementation, the depthdetermination module 116 generates the depth map D_(t) for each pixel ofthe captured image information I_(t) because that is the granularity inwhich the pattern shift Δ_(t) is generated.

At this stage, pattern generation functionality 118 analyzes the depthmap to generate a new pattern. More specifically, the pattern generationfunctionality 118 generates a new pattern P_(t) for a new time instancet based on a depth map D_(t−1) that has been captured in a prior timeinstance, e.g., t−1. In other implementations, however, the patterngeneration functionality 118 can generate a new pattern P_(t) based onthe n prior depth maps that have been generated, where n≧2. The patterngeneration functionality 118 can resort to this approach in cases inwhich there is quick movement in the scene 106. Additional explanationwill be provided at a later juncture of this description regarding theuse of plural depth maps to calculate P_(t). In the immediatelyfollowing explanation, however, it will be assumed that the patterngeneration functionality 118 generates P_(t) based on only thelast-generated depth map, namely D_(t−1).

The pattern generation functionality 118 includes a region analysismodule 120, a pattern assignment module 122, an intensity assignmentmodule 124, and a pattern composite module 126. To begin with, theregion analysis module 120 examines the depth map D_(t−1) to identifydepth regions in the depth map D_(t−1) that have surfaces of similardepths (with respect to the reference point, such as the projectordevice 104). In one implementation, for example, the region analysismodule 120 can use a region growing algorithm to identify M continuousand non-overlapping regions, each having a similar depth value. Theregion analysis module 120 can represent the depths of each depth regionas the average of the depths (of surfaces) within the region. That is,the depth region R(d₁) includes surfaces with an average depth d₁ withrespect to the reference point, the depth region R(d₂) includes surfaceswith an average depth d₂ with respect to the reference point, and so on.In FIG. 1, assume that the surfaces of the sphere 110 are encompassed bya first depth region, while the surfaces of the pyramid 112 areencompassed by a second depth region.

The region analysis module 120 then defines a set of masks B_(t)associated with the different depth regions, where B_(t)={B_(t,m)}_(m=1)^(M), and where B_(t,m) refers to an individual mask within a set of Mmasks. To perform this task, the region analysis module 120 maps thedepth regions that have been identified in the coordinate system of thecamera device 108 to the coordinate system of the projector device 104.This mapping can be derived from the calibration of the projector device104 and the camera device 108, described in Section D. The masksdemarcate different parts of the pattern P_(t) that is to be generated.For example, a first mask may demarcate a first part of the patternP_(t) that will be tailored to illuminate the sphere 110, while a secondmask may demarcate a second part of the pattern P_(t) that will betailored to illuminate the pyramid 112.

A pattern assignment module 122 assigns a different component pattern toeach depth region and its corresponding part of the pattern P_(t).(Insofar as the pattern P_(t) includes multiple components, it ishenceforth referred to as a composite pattern to distinguish it from itsconstituent component patterns.) Further, each component patternincludes features having a particular property, where differentcomponent patterns include features having different respectiveproperties. For example, in the detailed examples presented herein, eachcomponent pattern has speckle features of a particular size. In otherimplementations, each component pattern has code-bearing features havinga particular design (e.g., permutation), such as binary-coded featureshaving a particular design.

The pattern assignment module 122 assigns a component pattern to eachdepth region by consulting a mapping table. The mapping table mapsdifferent ranges of depths to different respective component patterns.That is, consider a depth region with an average depth of 1.5 m. Thepattern assignment module 122 can consult the mapping table to determinethe component pattern that is associated with this depth, where thatcomponent pattern has a particular feature property. The patternassignment module 122 will then assign the identified component patternto the part of the composite pattern P_(t) that is devoted to the depthregion in question.

An intensity assignment module 124 assigns an illumination intensity toeach depth region and its corresponding part of the composite patternP_(t). Generally, exposure E is related to illumination intensity q anddepth d according to the equation:

$E = {\frac{q}{d^{2}}.}$

Further, consider the case of a candidate pattern with a dynamic range[0, q₀], that is, with grayscale values ranging from 0 to q₀. Furtherassume that image information captured from the scene at reference depthd₀ is not underexposed (with respect to grayscale value 0) and notoverexposed (with respect to grayscale value q₀), but would beunderexposed and overexposed below and above this range, respectively.Proper exposure can be obtained for the same pattern with a scaleddynamic range [0, q] captured at depth d provided that:

$q = {\frac{d^{2}}{d_{0}^{2}}{q_{0}.}}$

The intensity assignment module 124 can leverage the above-describedprinciple by changing the dynamic intensity range of each projectedcomponent pattern to achieve a desired illumination intensity, based onthe average depth of a particular depth region. More specifically, theintensity assignment module 124 can assign an illumination intensity βto each depth region j (and each corresponding part of the compositepattern P_(t)) according to the following equation:

${\beta = {{\lambda \; \frac{d_{j}^{2}}{d_{0}^{2}}} + b}},$

where d_(j) corresponds to the average depth of the depth region j inquestion, d₀ corresponds to the reference depth, and λ and b areempirically-determined constants. In one merely representativeenvironment, λ is set to approximately 0.9 and b is set of approximately0.1.

The pattern composition module 126 constructs the new composite patternP_(t) based on the masks that have been generated by the region analysismodule 120, the component patterns that have been selected by thepattern assignment module 122, and the illumination intensities thathave been calculated by the intensity assignment module 124. Moreformally stated, in one implementation, the pattern composition module126 constructs the composite pattern P_(t) based on the followingequation:

${P_{t} = {\sum\limits_{m = 1}^{M}{P_{S,m}B_{t,m}\beta_{t,m}}}},{t > 1},$

where M refers to the number of parts in the composite pattern P_(t),P_(S,m) refers to the component pattern to be applied to the part m ofthe composite pattern P_(t) (which, in turn, corresponds to a particulardepth region in the depth map D_(t−1)), B_(t,m) refers to the mask thatdemarcates the part m, and β_(t,m) refers to the illumination intensityto be applied to the part m.

The projector device 104 projects this composite pattern P_(t) onto thescene 106, triggering the depth determination module 116 to captureanother depth map, and the pattern generation functionality 118 togenerate another composite pattern P_(t+1). In one implementation, thiscycle of computation repeats for every captured frame of imageinformation. However, in other implementations, the pattern generationfunctionality 118 can regenerate the composite pattern on a lessfrequent basis. For example, the pattern generation functionality 118may only generate a new composite pattern if it detects significantmovement of the objects within the scene 106.

Finally, set-up functionality 128 performs various roles in connectionwith setting up and initializing the ADSS 102. These tasks include:calibrating the projector device 104 and the camera device 108,capturing a reference image set I_(S), and capturing the first depthmap. Section D provides additional details regarding these preliminarytasks.

Advancing to FIG. 2, this figure provides a high-level conceptualdepiction of the operation of the pattern generation functionality 118of the ADSS 102. Assume that the region analysis module 120 hasidentified at least three depth regions associated with the scene 106and at least three corresponding masks. More specifically, the regionanalysis module 120 assigns a first mask B_(t,1) to a depth regionassociated with the sphere 110 and a second mask B_(t,2) to a depthregion associated with the pyramid 112. The region analysis module 120may also assign a background mask B_(t,M) to all other regions besidesthe depth region associated with the sphere 110 and pyramid 112.

The pattern assignment module 122 assigns a first component patternP_(S,1) to the part of the composite pattern demarcated by the firstmask B_(t,1), a second component pattern P_(S,2) to the part of thecomposite pattern demarcated by the second mask B_(t,2), and a thirdcomponent pattern P_(S,M) to the part of the composite patterndemarcated by the third mask B_(t,M). In this non-limiting example,these three component patterns are speckle patterns. Further, note thatthe speckle features of these three component patterns have differentrespective sizes; for instance, the speckle features in the firstcomponent pattern P_(S,1) have the largest size and the speckle featuresin the third component pattern P_(S,M) have the smallest size. Asmentioned above, in another example, each component pattern mayalternatively have a particular code-bearing design, differing from thecode-bearing designs of other component patterns.

Note that the star shapes in FIG. 2 represent the speckle features in ahigh-level conceptual form. In an actual implementation, the specklefeatures may have random (or pseudo-random) shapes and random (orpseudo-random) arrangements of those shapes. Further, the actual sizesof the speckle features may be different from those depicted in FIG. 2.Section B provides additional details regarding how speckle componentpatterns having different sized features may be generated.

The intensity assignment module 124 assigns a first illuminationintensity β_(t,1) to the part of the composite pattern demarcated by thefirst mask B_(t,1), a second illumination intensity β_(t,2) to the partof the composite pattern demarcated by the second mask B_(t,2), and athird illumination intensity β_(t,M) to the part of the compositepattern demarcated by the third mask B_(t,M).

The pattern composition module 126 produces the final composite patternP_(t) by superimposing the above-described parts of the compositepattern P_(t). As shown, the part of the composite pattern P_(t)associated with the sphere 110 includes the first component patternP_(S,1) and is illuminated by the first illumination intensity β_(t,1).The part of the composite pattern P_(t) associated with the pyramid 112includes the second component pattern P_(S,2) and is illuminated by thesecond illumination intensity β_(t,2), and the part of the compositepattern P_(t) associated with remainder of the depth map includes thethird component pattern P_(S,M) and is illuminated by the thirdillumination intensity β_(t,M). When projected, the different parts ofthe composite pattern P_(t) will effectively impinge different parts ofthe scene 106.

FIG. 3 shows a procedure 300 which represents an overview of one mannerof operation of the ADSS 102 of FIG. 1. FIG. 4 shows a procedure 400that represents a more detailed explanation of the operation of the ADSS102. Since the operation of the ADSS 102 has already been described inthe context of FIG. 1, the explanation of FIGS. 3 and 4 will serve as asummary.

In block 302 of FIG. 302, the ADSS 102 receives image information fromthe camera device 108. The image information represents a scene that hasbeen illuminated by a composite pattern. In block 304, the ADSS 102generates a depth map based on the captured image information. In block306, the ADSS 102 generates a new composite pattern having parts thatare selected based on different depth regions within the depth map. Inblock 308, the ADSS 102 instructs the projector device 104 to projectthe new composite pattern onto the scene 106. This process repeatsthroughout the image capture session. Overall, the depth determinationmodule 116 and the pattern generation functionality 118 cooperativelyinteract to reduce defocus blur, underexposure, and overexposure in thecaptured image information.

Advancing to FIG. 4, in block 402, the ADSS 102 performs various set-uptasks, such as calibrating the camera device 108 and the projectordevice 104. In block 404, the ADSS 102 generates an initial depth map.Section D (below) provides details regarding blocks 402 and 404.

In block 406, the ADSS 102 identifies depth regions in the depth map,and generates masks corresponding to the depth regions. In block 408,the ADSS 102 assigns component patterns to the depth regions. In block410, the ADSS 102 assigns illumination intensities to the depth regions.In block 412, the ADSS 102 composes a new composite pattern P_(t) basedon the masks, component patterns, and illumination intensities that havebeen determined in blocks 406-410. Blocks 406-412 correspond to block306 of FIG. 3.

In block 414 (corresponding to block 308 of FIG. 3), the ADSS 102instructs the projector device 104 to project the new composite patternP_(t) onto the scene 106. In block 416 (corresponding to blocks 302 and304 of FIG. 3), the ADSS 102 receives information from the camera device108 and generates a new depth map based on the image information. In thecontext of the subsequent generation of yet another new compositepattern P_(t+1), the depth map D_(t) that is generated in block 416 canbe considered the “prior” depth map.

More specifically, recall that depth determination module 116 computesthe pattern shift between the captured image information I_(t) and thereference image I_(t)′, and then uses the triangulation principle tocompute the depth map. The reference image I_(t)′ can be generated basedon the equation:

${I_{t}^{\prime} = {\sum\limits_{m = 1}^{M}{I_{S,m}B_{t,m}^{\prime}}}},{t > 1.}$

I_(S,m) refers to a component reference image in a reference image setI_(S). Section D describes one way in which the reference image setI_(S) can be calculated. B_(t,m)′, refers to a resized version of themask B_(t,m), determined by mapping between the projector devicecoordinates and the camera device coordinates. This mapping can bederived from the calibration of the projector device 104 and the cameradevice 108, which is also described in Section D.

As a final point, the ADSS 102 is described above as generating a newcomposite pattern P_(t) based on only the last-generated depth mapD_(t−1). But, as said, the ADSS 102 can alternatively generate the newcomposite pattern P_(t) based on the n last depth maps, where n≧2. Inthis case, the region analysis module 120 can analyze the plural depthmaps to predict, for each object in the scene 106, the likely depth ofthat object at the time of projection of the composite pattern P_(t).The region analysis module 120 can perform this task by extending thepath of movement of the object, where that path of movement is exhibitedin the plural depth maps.

B. Deriving the Component Patterns

FIG. 5 shows a pattern set generation module (PSGM) 502 for generating aset of component patterns P_(s). As described in Section A, thecomponent patterns include features having different respectiveproperties. For example, as shown in FIG. 2, the component patterns mayinclude speckle features having different respective sizes. A data store504 stores the set of component patterns P_(s). The data store 504 isaccessible to the pattern generating functionality 118.

FIG. 6 shows one procedure 600 for creating a set of component specklepatterns P_(s) using a simulation technique. Generally, speckle occurswhen many complex components with independent phase are superimposed.FIG. 6 simulates this phenomenon using the Discrete Fourier Transform(DFT). FIG. 6 will be explained below with respect to the generation ofa single component pattern having a particular speckle feature size. Butthe same process can be repeated for each component speckle pattern inP_(s), having its associated speckle feature size.

Beginning with block 602, the PSGM 502 produces an N by N random phasematrix Θ, defined as:

${\Theta \left( {x,y} \right)} = \left\{ \begin{matrix}{\theta_{U{\lbrack{0,1}\rbrack}},} & {{1 \leq x},{y \leq {N/K}}} \\{0,} & {{otherwise}.}\end{matrix} \right.$

In this equation, θ_(U[0,1]) denotes a random number taken from auniform distribution on the interval [0,1], and K represents a factor ofN that ultimately will determine the size of the speckle features in thecomponent pattern being generated.

In block 604, the PSGM 502 produces a random complex matrix A, definedas:

A(x,y)=e ^(2πiΘ(x,y)), 1≦x,y≦N.

In block 606, the PSGM 502 applies a 2D-DFT operation on A to yieldanother random complex matrix Z, defined as:

${{Z\left( {x,y} \right)} = {\sum\limits_{u = 1}^{N}{\sum\limits_{v = 1}^{N}{{A\left( {u,v} \right)}^{{- 2}\pi \; {{{({{ux} + {vy}})}}/N}}}}}},{1 \leq x},{y \leq {N.}}$

In block 608, the PSGM 502 generates the component pattern, representedby speckle signal S, by calculating the modulus square of each complexelement in Z, to yield:

S(x,y)=|Z(x,y)|², 1≦x,y≦N.

In one merely representative environment, the PSGM 502 performs theprocedure 600 of FIG. 6 for K=4, 8, 16, and 32, resulting in fourcomponent patterns. The component pattern for K=4 will have the smallestspeckle feature size, while the component pattern for K=32 will have thelargest speckle feature size.

As mentioned above, other implementations of the ADSS 102 can usecomponent patterns having other types of features to produce structuredlight when projected onto the scene 106. In those contexts, the PSGM 502would generate other types of component patterns, such as componentpatterns having code-bearing features.

C. Deriving the Mapping Table

FIG. 7 represents a procedure 700 for deriving a mapping table. Asdescribed in Section A, the mapping table correlates different depthranges in a scene with different component patterns. The componentpattern that maps to a particular depth range corresponds to theappropriate component pattern to project onto surfaces that fall withinthe depth range, so as to reduce the effects of defocus blur,underexposure, and overexposure.

In block 702, a defocus model is derived to simulate the effects of blurat different depths. One manner of deriving the defocus model isdescribed below with reference to the representative lens and raydiagram of FIG. 8. More specifically, FIG. 8 shows a projector lens 802used by the projector device 104, and a camera lens 804 used by thecamera device 108. In this merely representative configuration, theprojector lens 802 is vertically disposed with respect to the cameralens 804. Further assume that the focal planes of both the projectordevice 104 and the camera device 108 correspond to focal plane 806,which occurs at distance d₀ with respect to the projector lens 802 andthe camera lens 804. The projector device 104 has an aperture A₁, whilecamera device 108 has an aperture A₂. A projector device emitter 808projects light into the scene 106 via the projector lens 802, while acamera device CCD 810 receives light from the scene 106 via the cameralens 804. The camera device CCD 810 is located at a distance of l₀ fromthe camera device lens 804. A point from the focal plane 806 (atdistance d₀) will converge on the camera device CCD 810.

Consider the case of an object plane at distance d. Because d is notcoincident with d₀, a blur circle having diameter C, on the objectplane, is caused by defocus of the projector device 104. Using theprinciple of similar triangles, the projector device's blur circle C canbe expressed as:

$C = {A_{1}{\frac{{d_{0} - d}}{d_{0}}.}}$

Moreover, a point from the object plane at distance d will convergebeyond the camera device CCD 810, at a distance l to the camera devicelens 804. Thus, another blur circle, of diameter c, on the camera deviceCCD 810, is caused by defocus of the camera device 108. Using theprinciple of similar triangles, the camera device's blur circle c can beexpressed as:

$c = {{\frac{l_{0}}{d}C} + {A_{2}{\frac{{l - l_{0}}}{l}.}}}$

Assume that the focal length of the camera device 108 is F. According tothe lens equation:

$\frac{1}{F} = {{\frac{1}{l_{0}} + \frac{1}{d_{0}}} = {\frac{1}{l} + {\frac{1}{d}.}}}$

Using the above equation, c can be alternatively expressed as:

$c = {\frac{\left( {A_{1} + A_{2}} \right)F}{d_{0} - F}{\frac{{d_{0} - d}}{d}.}}$

The defocus bur can be modeled as an isotropic, two-dimensional Gaussianfunction. The standard deviation σ of this function is proportional tothe blur circle c. Since A₁, A₂, d₀, and F are all constants, thestandard deviation can be expressed as:

$\sigma = {{k{{\frac{d_{0}}{d} - 1}}} + {\sigma_{0}.}}$

Returning to FIG. 7, in block 702, the parameters k and σ₀ in the aboveequation are estimated. To perform this task, the projector device 104can project a pattern with repetitive white dots as an impulse signal.The camera device 108 can then capture the resultant image informationfor different object planes at different depths. The diameter of blur(measured in pixels) exhibited by the image information can then becomputed for different depths.

For example, FIG. 9 shows, for one illustrative environment,measurements of the diameter of blur with respect to different depths,where the focal plane is at a distance of 2.0 m. The least square methodcan then be used to fit a curve to the observations in FIG. 9. Thatcurve yields the values of k and σ₀. In one merely representativeenvironment, k is estimated to be approximately 4.3 and σ₀ is estimatedto be approximately 1.0. Note that the diameter of blur converges to alow value on the far side of the focal plane (at distances greater than2.0 m). Thus, it is not necessary to consider defocus blur at those fardistances.

In block 706 of FIG. 7, the defocus model that has just been derived isused to induce blur in different component patterns, for differentdepths. Then, a measure is generated that reflects how closely theoriginal (un-blurred) component pattern matches its blurred counterpart.For example, that measure can define the number of elements in theoriginal component pattern that were successfully matched to theirblurred counterparts. FIG. 10 shows the representative outcome of thisoperation, in one illustrative environment, for depths ranging from 0.4m to 2.0 m, and for speckle feature sizes of K=1 (the smallest) to K=32(the largest).

In block 708 of FIG. 7, component patterns are chosen for differentdepth ranges. Different considerations play a role in selecting acomponent pattern for a particular depth range. The considerationsinclude at least: (a) the matching accuracy (as represented by FIG. 10);(b) local distinguishability; and (c) noise.

Matching Accuracy. Matching accuracy refers to the ability of acomponent pattern to effectively reduce defocus blur within a particulardepth range. In this regard, FIG. 10 indicates that larger specklefeatures perform better than smaller speckle features as the objectplane draws farther from the focal plane 806 (and closer to theprojector lens 802 and camera lens 804). Consider, for example, a firstcomponent pattern having a first feature size K₁, and a second componentpattern having a second feature size K₂, where K₁>K₂. By consideringmatching accuracy alone, the first component pattern is appropriate fora first depth range d₁ and the second component pattern is appropriatefor a second depth range d₂, providing that d₁<d₂.

Local distinguishability. Local distinguishability refers to the abilityof the ADSS 102 to obtain accurate depth readings around objectboundaries. With respect to this consideration, smaller speckle featuresperform better than larger speckle features.

Noise. Noise refers to the amount of noise-like readings produced whencapturing image information. With respect to this consideration, smallerspeckle features induce more noise than larger speckle features.

A system designer can take all of the above-described factors intoaccount in mapping different component patterns to respective depthranges, e.g., by balancing the effects of each consideration against theothers. Different mappings may be appropriate for differentenvironment-specific objectives. In some cases, the system designer mayalso wish to choose a relatively small number of component patterns tosimplify the operation of the ADSS 102 and make it more efficient.

FIG. 11 shows one mapping table which maps component patterns(associated with different speckle feature sizes) with different depthranges. In block 710 of FIG. 7, the mapping table is stored in a datastore 1102. The pattern assignment module 122 has access to the mappingtable in the data store 1102.

D. Preliminary Operations

Returning to FIG. 4, in block 402, the system designer can calibrate theprojector device 104 and the camera device 108. This ultimately providesa way of mapping the coordinate system of the projector device 104 tothe coordinate system of the camera device 108, and vice versa. Knownstrategies can be used to perform calibration of a structured lightsystem, e.g., as described in Jason Geng, “Structured-light 3D surfaceimaging: a tutorial,” Advances in Optics and Photonics, Vol. 3, 2011,pp. 128-160. For example, block 402 may entail calibrating the cameradevice 108 by capturing image information of a physical calibrationobject (e.g., a checkerboard pattern) placed at known positions in thescene 106, and calibrating the camera device 108 based on the capturedimage information. Calibration of the projector device 104 (which can betreated as an inverse camera) may entail projecting a calibrationpattern using the projector device 104 onto a calibration plane,capturing the image information using the calibrated camera device 108,and calibrating the projector device 104 based on the captured imageinformation.

Block 402 also involves generating the reference image set I_(S), whereI_(S)={I_(S,m)}_(m=1) ^(M). Each component reference image I_(S,m) inthe set is produced by projecting a particular component pattern (havinga particular property) onto a reference plane at a known depth D₀,oriented normally to the light path of the projector device 104. Thecamera device 108 then captures image information of the scene 106 toyield I_(S,m).

In block 404, the set-up functionality 128 determines an initial depthmap. In one approach, the set-up functionality 128 can instruct theprojector device 104 to successively project different componentpatterns in the set P_(S), at different illumination intensities. Thedepth determination module 116 can then provide depth maps for eachcombination of P_(s,m) and β_(s,m). For each point in the scene, theset-up functionality 128 then selects the depth value that occurs mostfrequently within the various depth maps that have been collected. Thisyields the initial depth map when performed for all points in the scene.

E. Representative Computing Functionality

FIG. 12 sets forth illustrative computing functionality 1200 that can beused to implement any aspect of the functions described above. Forexample, the computing functionality 1200 can be used to implement anyaspect of the ADSS 102. In one case, the computing functionality 1200may correspond to any type of computing device that includes one or moreprocessing devices. In all cases, the computing functionality 1200represents one or more physical and tangible processing mechanisms.

The computing functionality 1200 can include volatile and non-volatilememory, such as RAM 1202 and ROM 1204, as well as one or more processingdevices 1206 (e.g., one or more CPUs, and/or one or more GPUs, etc.).The computing functionality 1200 also optionally includes various mediadevices 1208, such as a hard disk module, an optical disk module, and soforth. The computing functionality 1200 can perform various operationsidentified above when the processing device(s) 1206 executesinstructions that are maintained by memory (e.g., RAM 1202, ROM 1204,and/or elsewhere).

More generally, instructions and other information can be stored on anycomputer readable storage medium 1210, including, but not limited to,static memory storage devices, magnetic storage devices, optical storagedevices, and so on. The term computer readable storage medium alsoencompasses plural storage devices. In all cases, the computer readablestorage medium 1210 represents some form of physical and tangibleentity.

The computing functionality 1200 also includes an input/output module1212 for receiving various inputs (via input modules 1214), and forproviding various outputs (via output modules). One particular outputmechanism may include a presentation module 1216 and an associatedgraphical user interface (GUI) 1218. The computing functionality 1200can also include one or more network interfaces 1220 for exchanging datawith other devices via one or more communication conduits 1222. One ormore communication buses 1224 communicatively couple the above-describedcomponents together.

The communication conduit(s) 1222 can be implemented in any manner,e.g., by a local area network, a wide area network (e.g., the Internet),etc., or any combination thereof. The communication conduit(s) 1222 caninclude any combination of hardwired links, wireless links, routers,gateway functionality, name servers, etc., governed by any protocol orcombination of protocols.

Alternatively, or in addition, any of the functions described in thepreceding sections can be performed, at least in part, by one or morehardware logic components. For example, without limitation, thecomputing functionality can be implemented using one or more of:Field-programmable Gate Arrays (FPGAs); Application-specific IntegratedCircuits (ASICs); Application-specific Standard Products (ASSPs);System-on-a-chip systems (SOCs); Complex Programmable Logic Devices(CPLDs), etc.

In closing, the description may have described various concepts in thecontext of illustrative challenges or problems. This manner ofexplanation does not constitute an admission that others haveappreciated and/or articulated the challenges or problems in the mannerspecified herein.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

What is claimed is:
 1. An adaptive depth sensing system, implemented bycomputing functionality, comprising: a depth determination moduleconfigured to generate a depth map based on image information receivedfrom a camera device; and pattern generation functionality configuredto: analyze the depth map to provide an analysis result; generate acomposite pattern based on the analysis result; and instruct a projectordevice to project the composite pattern onto a scene, the depthdetermination module and pattern generation functionality configured torepeatedly generate depth maps and composite patterns, respectively, thedepth determination module and pattern generation functionality beingimplemented by the computing functionality.
 2. The adaptive depthsensing system of claim 1, wherein the depth determination module andthe pattern generation functionality cooperatively interact to reduceoccurrence of at least one of: defocus blur in the image information;underexposure in the image information; and overexposure in the imageinformation.
 3. The adaptive depth sensing system of claim 1, whereinsaid pattern generating functionality includes a region analysis modulethat is configured to: identify depth regions in the depth map, eachdepth region corresponding to a region of the scene with similar depthswith respect to a reference point; and identify a set of masksassociated with the depth regions.
 4. The adaptive depth sensing systemof claim 3, wherein said pattern generating functionality also includesa pattern assignment module that is configured to assign componentpatterns to the respective depth regions, each component patternincluding features having a particular property, and different componentpatterns including features having different respective properties. 5.The adaptive depth sensing system of claim 3, wherein said patterngenerating functionality also includes an intensity assignment modulethat is configured to assign an illumination intensity β for a depthregion j in the depth map, as given by:${\beta = {{\lambda \; \frac{_{j}^{2}}{_{0}^{2}}} + b}},$ whered_(j) is a representative depth of the depth region j in the depth map,d₀ is a reference depth, and λ and b are constants.
 6. The adaptivedepth sensing system of claim 1, wherein the composite pattern is givenby: ${P_{t} = {\sum\limits_{m = 1}^{M}{P_{S,m}B_{t,m}\beta_{t,m}}}},$where P_(t) is the composite pattern, M is a number of parts in thecomposite pattern, P_(S,m) is a component pattern for application topart m, B_(t,m) is a mask which defines the part m, β_(t,m) is anillumination intensity for application to part m, and t is time.
 7. Amethod, performed by computing functionality, for generating a depthmap, comprising: receiving image information from a camera device, theimage information representing a scene that has been illuminated with acomposite pattern; generating a depth map based on the imageinformation, the depth map having depth regions, each depth regioncorresponding to a region of the scene having similar depths withrespect to a reference point; generating a new composite pattern havingparts that are selected based on the respective depth regions in thedepth map; instructing a projector device to project the new compositepattern onto the scene; and repeating said receiving, generating a depthmap, generating a new composite pattern, and instructing at least onetime.
 8. The method of claim 7, wherein said generating a new compositepattern includes: identifying the depth regions in the depth map; andidentifying a set of masks associated with the respective depth regions.9. The method of claim 7, wherein said generating a new compositepattern includes assigning component patterns to the respective depthregions, each component pattern including features having a particularproperty, and different component patterns including features havingdifferent respective properties.
 10. The method of claim 9, wherein theparticular property associated with each component pattern is featuresize.
 11. The method of claim 10, wherein the features associated witheach component pattern correspond to speckle features of a particularsize.
 12. The method of claim 9, wherein the particular propertyassociated with each component pattern is code-bearing design.
 13. Themethod of claim 9, wherein said assigning involves assigning a firstcomponent pattern having a first feature size K₁ for a first depthregion d₁ and a second component pattern having a second feature size K₂for a second depth region d₂, where K₁>K₂ if d₁<d₂.
 14. The method ofclaim 9, wherein the component patterns are produced using a simulationtechnique.
 15. The method of claim 7, wherein said generating a newcomposite pattern includes assigning illumination intensities to therespective depth regions.
 16. The method of claim 15, wherein anillumination intensity β for a depth region j is given by:${\beta = {{\lambda \; \frac{d_{j}^{2}}{d_{0}^{2}}} + b}},$ whered_(j) is a representative depth of the depth region j in the depth map,d₀ is a reference depth, and λ and b are constants.
 17. The method ofclaim 7, wherein the new composite pattern is given by:${P_{t} = {\sum\limits_{m = 1}^{M}{P_{S,m}B_{t,m}\beta_{t,m}}}},$where P_(t) is the new composite pattern, M is a number of parts in thenew composite pattern, the parts being associated with differentrespective depth regions, P_(S,m) is a component pattern for applicationto part m, B_(t,m) is a mask which defines the part m, β_(t,m) is anillumination intensity for application to part m, and t is time.
 18. Themethod of claim 7, wherein the new composite pattern is formed based ona consideration of n previously generated depth maps, where n≧2.
 19. Acomputer readable storage medium for storing computer readableinstructions, the computer readable instructions providing an adaptivedepth sensing system when executed by one or more processing devices,the computer readable instructions comprising: logic configured toreceive image information from a camera device, the image informationrepresenting a scene that has been illuminated with a composite pattern;logic configured to generate a depth map based on the image information;logic configured to identify depth regions in the depth map, each depthregion corresponding to a region of the scene with similar depths withrespect to a reference point; logic configured to identify a set ofmasks associated with the depth regions; logic configured to assigncomponent patterns to the respective depth regions, each componentpattern including features having a particular property, and differentcomponent patterns including features having different respectiveproperties; logic configured to assign illumination intensities to therespective depth regions; logic configured to produce a new compositepattern based on the masks, component patterns, and illuminationintensities; and logic configured to instruct a projector device toproject the new composite pattern onto the scene.
 20. The computerreadable storage medium of claim 19, wherein the new composite patternis given by:${P_{t} = {\sum\limits_{m = 1}^{M}{P_{S,m}B_{t,m}\beta_{t,m}}}},$where P_(t) is the new composite pattern, M is a number of parts in thenew composite pattern, the parts being associated with differentrespective depth regions, P_(S,m) is a component pattern for applicationto part m, B_(t,m) is a mask which defines the part m, β_(t,m) is anillumination intensity for application to part m, and t is time.